Systems and Methods for Testing Skills Capability Using Technologically-Enhanced Questions in a Computerized Environment

ABSTRACT

Systems and processes are provided for analyzing a test taker&#39;s knowledge, skills, and cognitive function. The systems and processes include smart questions (i.e., SmartItems) programmed with coded variables, or for multiple choice varieties, variable sets of options to cover an entire skill set being tested. The systems and processes generate questions in real time during administration of the test, by formulating and displaying, to a test taker, random combinations of a question stem and randomly-selected question components. This smart question content is unpredictably and randomly presented, as representations of the skill to be tested. The systems and processes analyze the test taker&#39;s knowledge, skills, and cognitive function about the entire domain of the skill, rather than a single fact. The systems and processes provide security to test administration.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the priority and benefit of U.S. ProvisionalPatent Application Ser. No. 62/598,037, titled “Systems and Methods forTesting Skills Capability Using Technologically Enhanced Questions in aComputerized Environment,” filed on Dec. 13, 2017, the entirety of whichis incorporated herein by reference.

FIELD OF THE INVENTION

The invention relates generally to the field of testing an individualtest taker's knowledge, skills, and the cognitive processes by which theindividual applies such knowledge and skills by usingtechnologically-enhanced questions with computerized systems andmethods. Specifically, the systems and methods of the invention providea computer-based test platform that delivers discrete,technologically-enhanced test questions useful for analyzing acquiredknowledge and comprehension by a test taker of a tested skill. Thetested skill is identical for each test taker, but test questions aresystematically formulated and presented in pairings of a question orstatement with a response mode so the combination is entirely unique toa test-taker. This provides test security and quality of skillsmeasurement, along with lower costs for initial development andlong-term maintenance.

BACKGROUND

Over time, the testing models by which an individual's knowledge onspecific subjects have been tested has generally remained the same. Forexample, a subject matter expert, teacher, or professor prepares aseries of test questions and corresponding answers, which test knowledgeon specific subjects. The questions and answers are organized into asingle test of the same questions which is administered to multipletest-takers. Such fixed questions are invariant, and can be easilycopied, stolen, memorized, shared, sold, etc. Furthermore, pre-knowledgeof those items can be reliably used to prepare for tests and to providean unfair advantage on tests, i.e., the ability to cheat.

Since the early 1980s, computer technology has been used in limited waysin the field of test development and administration. For example,computer-based testing is commonly used for various subject matters,such as, standardized tests for professional certification, e.g., theNational Council Licensure Examination (NCLEX) and the Patent BarExamination (Patent Bar); for graduate school admissions, e.g., theGraduate Management Admission Test (GMAT), the Graduate RecordExaminations (GRE), and the Medical College Admission Test (MCAT); andfor other testing and certifications at various levels of education andlicensure. It has become routine for test administrators to offeryear-round, computer-based testing at test centers across the countryand throughout the world. While technology has improved the convenienceof test-taking, the applied testing model generally performs in many ofthe same ways as traditional paper tests. Even with computerizedadaptive testing (CAT), the adaptive nature of the test focuses on acomputer system re-estimating a test taker's ability, based on the testtaker's previous answers and the difficulty of questions. In otherwords, CAT learns about a test taker's knowledge based on answers toquestions, and then presents to the test taker subsequent pre-formulated(static) questions from a test bank that correlate with the test taker'sknowledge. CAT does not implement smart questions (i.e., SmartItems),randomly formulated in real time during test administration, to analyzea test taker's knowledge retention or other skills, cognitive orbehavioral, as provided by the invention.

Traditional testing models have at least three significant andrecognized flaws.

First, many testing models use questions or items that are limited incapability and encourage improper motivation to prepare including, forexample, rote memorization. Indeed, an applicant sitting for the PatentBar Examination could simply memorize the black letter rules, i.e., thePatent Rules (37 Code of Federal Regulations) and/or relevant chaptersof the Manuel of Patent Examining Procedure (MPEP), rather than obtainthe knowledge necessary to apply the rules of practice and procedure.While rote memorization can lead the test taker to passage of a test,the test taker will likely lack the fundamental cognitive skillsnecessary to apply the memorized information.

Second, many tests can be copied or stolen, which provide opportunitiesfor individuals to obtain pre-test knowledge of the test, and then usethat pre-test knowledge to cheat on a test. Fixed test questions andanswers can be quickly learned solely to pass the test. Also, suchquestions and answers can be easily shared or sold over the Internet.This is a significant problem, particularly in the area of professionalcertification tests, but has also been the foundation of many recentteacher-based scandals in statewide K-12 educational testing. Moreover,certain test administration bodies require test takers to sign anon-disclosure agreement in an attempt to safeguard the confidentialityof the questions and answers presented on a standardized test, butviolations of such agreements can be difficult to identify and requirecostly enforcement of rights on behalf of the test administrationbodies.

Third, traditional testing models can be inequitable to some populationsof test takers. Testing can be difficult for some test takers, whilebeing easier for others, even though test takers have the same access tothe content being tested, as well as the test itself. This differencehas nothing to do with the content of the test, but rather theindividuals taking the test, their test taking experience,socio-economic status, and physical makeup. For example, some affectedindividuals include the visually impaired, others are those withlearning disabilities or language comprehension issues. Also, some testtakers are more capable of learning tricks and cues for traditionaltesting models, giving those test takers a significant unfair advantage.

Consequently, there is a need for computer-based test systems andmethods that: (a) use test items to measure an individual's knowledge,skills, and cognitive processes instead of his/her ability to simplymemorize content for a test; (b) are secure from being copied,reproduced, and distributed; and (c) equalize the fairness of testadministration for test takers.

SUMMARY

Illustrative and alternative embodiments of a computer-based testplatform are provided. The platform delivers discrete,technologically-enhanced test questions for analyzing a test taker'sknowledge and comprehension of a tested skill, whereby the tested skillis identical for each test taker, but systematically formulated andpresented in pairings of a stem and response modes, in real time duringtest administration. The stem is comprised of questions or statementswith alternatives for eliciting test taker responses. Each stem may beentirely unique for each test-taker, or the stem and response modecombination must be unique for each test taker, thus providing testsecurity and enhanced measurement of a test taker's knowledge, skills,and cognitive function.

The discrete, technologically-enhanced test questions either: (1)comprise question stems combined with randomly-selected variables tiedto the question stems to create unique questions that holistically testa skill, or (2) comprise question stems linked to additional questionstems, wherein each individual question stem may be combined withrandomly-selected variables tied to the question stems to create uniquequestion constellations. Furthermore, the test-taker's inputs may takenumerous response modes. For example, the question may present thevariations as differing constellations of answer options, presentedsimultaneously (as with Discrete Option Multiple Choice testing or DOMC)or sequentially. In testing administration, the systems and methods ofthe invention process a question in a manner that tests the same skillwith multiple test takers, but randomly generates and presents uniqueand discrete questions or statements and/or combinations of responsemodes and their format to each test taker. Because each question israndomly generated in real time during test administration, and isdiscrete and unique for each test taker, there is no advantage forsomeone to show or tell another person the questions or answers thatwere on the test. This mitigates cheating and ensures secure testadministration.

By way of an exemplary, non-limiting embodiment of the invention, theinvention provides a solution to the circumstance in which changes inthe question stem are combined with a variation in how the response isrequired, including how the answer options are presented. Here, if ateacher wanted to test an individual's ability to add two-digit numbers,then the partial question stem could be prepared as “[Variable1]+[Variable 2]=”. There are 89 variables that can be randomly selected,i.e., integers between 10 and 99, and there are 7,921 unique questioncombinations, i.e., 89×89=7,291. The scope of variables and answersmakes cheating very difficult. In this example, the method for the testtaker to respond could be by selecting the correct answer, or answers,from a list of answer options, or by seeing the answer options one at atime, with “Yes” and “No” answer options, and selecting “Yes” when thecorrect option or options is/are displayed, and “No” when the incorrectoption or options is/are displayed, or by supplying a response by typingwords or characters from a keyboard.

By way of another exemplary, non-limiting embodiment of the invention,the invention provides a smart question having a complete stem, and thevariability in the options can be created by simply selecting optionsfrom dozens, hundreds, thousands, or more, in an option pool. Forexample, the stem could be, “Identify the mammal.” and the optionsdisplayed to the test taker could be 4 (i.e., one correct and 3incorrect) selected from a list of 500 actual mammals and 5,000 actualnon-mammals. The total number of potential options allows the designerto cover the skill completely, and the options presented to the testtaker (whether presented sequentially or all at once) permits areasonable response. Furthermore, the number of correct answers is notrestricted to one. Indeed, there could be multiple mammals presented tothe test taker.

In one embodiment, the stem, variable, or answer can be an image thatdoes not contain any letters or numbers. In another embodiment referredto as a hot-area SmartItem, the areas of an image can comprise differentstem, variable, and response elements. For such a hot-area SmartItem,the test taker's response or answer can take the form of clicking withina specific area of the image presented to the test taker.

By way of a further exemplary, non-limiting embodiment of the invention,the system of the invention comprises a computer processor, a storagedevice for storing and recalling stored data, a display (i.e., userinterface), an applicant file, and a test file. Using the system, thetest taker signs into his/her account using unique identifyingcredentials, which provide access to the electronically-administeredtest that the test taker will be taking. For each test question that isadministered by the system to the test taker, the processor calls up aquestion having at least one variable coded into the question stem, andthen randomly calls up the at least one value of the variable that iscoded or linked to the question stem, both from one or more storagedevices, e.g., database(s), in communication with the system. Theprocessor then combines the randomly selected variable with the questionstem, in real time at test administration, to create the test questionpresented to the test taker. The processor also identifies the answer(s)that is/are linked to the randomly-created question in the storagedevice, and, optionally, at least one incorrect answer to the testquestion, all from the storage device or multiple storage devices. Theprocessor transmits the test question, the correct answer (i.e., key),and at least one answer option (i.e., correct or incorrect) to thesoftware application. The software application then displays thequestion and randomly either the correct or incorrect answer option, andasks the applicant if the answer or incorrect answer option is thecorrect answer. For each answer option that is displayed, the test takerindicates using “Yes” or “No” answers, whether he or she believes theanswer option to be correct or not, respectively. Depending on thecorrectness of the test taker response, the software application either:ends the question, or provides another answer option. The system canrepeat this process until either: the test taker properly selects thecorrect answer, or wrongly selects an incorrect option as being thecorrect answer. In either instance, the software application may thendisplay a new question. Often, such questions will present a largenumber of variations to the stem, and the response alternative will besomething other than a set of multiple-choice options. For example, thequestion could randomly select a picture of an animal from thousands inthe database, present it, and then ask the test taker to type in thename of the animal pictured in the response box provided.

In embodiments of the invention, the potential answers need not bepresented in a multiple choice format. Instead, the test taker cananswer in the form of: a short typed response, a longer typed responsetypical for an essay, the re-ordering of a list, the highlighting of thepassage in text, or any other form or format.

The method and non-transitory computer readable medium integrated withinthe system is comprised of, first, confirming that a test taker isregistered to take a test. The test commences and questions aregenerated. Each question is comprised of a question stem and a responsealternative. The selected question stem may have at least one variablecoded into the question stem, and randomly-selected variable tied to thequestion stem in the database. The response alternatives can also havevariability coded into either the number of available options, or thenumber of options ultimately presented. The question stem and thevariable are then presented with fixed alternatives, or combined withthe random response alternative variations to create the test questionpresented to a test taker. Similarly, the response alternative variationcan be combined with a fixed stem or a stem with at least one variableto create the question presented to the test taker. To clarify, theresponse alternative variations can be created from hundreds, thousands,or more, possible answer options, one or more of which can be correctoptions. Finally, the test question is presented to the test taker fortesting purposes. The test taker reads the stem, and is then presentedwith a variable set of options, at least one of which is a correctoption, or another way to respond, such as typing in the response. In anexample with the response alternative provided in the DOMC format, ifthe test taker is shown the stem of the test question and an incorrectanswer option and correctly responds that the combination is notcorrect, then the system can then pull from the data storage device andpresent the correct option or a second incorrect option, and, again, askif the combination is correct. If the test taker is shown the questionand the answer, and responds that the combination is correct, then thetest taker can be notified that the question has been answeredcorrectly. Conversely, if the test taker is shown the questions and anincorrect answer, and the test taker responds that the answer iscorrect, then the system will score the item as incorrectly answered andmove on to the next question in the test.

Optionally, if the test taker is shown the question and the answer inmultiple choice format and incorrectly responds that the combination isnot correct, then the system can alert the test taker of the incorrectlyselected answer to the question, or the system can provide theindividual with a new question and incorrect answer combination so as toprotect the identity of the correct answer. Such an approach providesfurther testing security.

In another embodiment, the system is capable of determining whether thetest taker's answer is correct or not. For example, the system candetermine if the reply submitted by the test taker is linked to both thequestion stem and the variable. If the reply is linked to both, then thesystem confirms that the reply is a correct answer, and the softwareapplication can display the number of correct responses.

Optionally, a time limit can be set for a test question, in which a testtaker must provide a reply to a SmartItem, or, if no reply is provided,then the test question is tallied as incorrect. In another embodiment, atest taker's replies can be limited for each SmartItem.

These and other features, aspects, and advantages of the invention willbecome better understood with reference to the following description andthe appended claims.

BRIEF DESCRIPTION OF THE FIGURES

Additional aspects, features, and advantages of the invention, as to itssystem, structure, components, configuration, and operability will beunderstood and become more readily apparent when the invention isconsidered in light of the following description of the figures made inconjunction with the accompanying drawings, wherein:

FIG. 1 shows the flow of information between components of the system ofthe invention.

FIG. 2 shows the hardware components of the system of the invention.

FIG. 3 shows a flow chart of an embodiment of the creation of aSmartItem of the invention.

FIG. 4 shows a flow chart of an embodiment of the creation of aSmartItem of the invention.

FIG. 5A shows a flow chart of an embodiment of the creation of aSmartItem of the invention.

FIG. 5B is a continuation of the flow chart of FIG. 5A depicting anembodiment of the creation of a SmartItem of the invention.

FIG. 6A shows an embodiment of a DOMC format SmartItem of the inventionas it will be displayed during a test with the variable identified bythe shading proceeding up and to the right, the stem identified by theshading proceeding up and to the right, and the answer identified by thecross hatching.

FIG. 6B shows an embodiment of a text entry format SmartItem of theinvention as it will be displayed during a test with the variableidentified by the shading proceeding up and to the right, the stemidentified by the shading proceeding up and to the right, and the answeridentified by the cross hatching.

FIG. 6C shows an embodiment of a Boolean format SmartItem of theinvention as it will be displayed during a test with the variableidentified by the shading proceeding up and to the right, the stemidentified by the shading proceeding up and to the right, and the answeridentified by the cross hatching.

FIG. 6D shows an embodiment of a Jeopardy® format SmartItem of theinvention as it will be displayed during a test with the variableidentified by the shading proceeding up and to the right, the stemidentified by the shading proceeding up and to the right, and the answeridentified by the cross hatching.

FIG. 6E shows an embodiment of a hot area format SmartItem of theinvention as it will be displayed during a test with the variableidentified by the shading proceeding up and to the right, the stemidentified by the shading proceeding up and to the right, and the answeridentified by the cross hatching.

FIG. 6F shows an embodiment of a list entry format SmartItem of theinvention as it will be displayed during a test with the variableidentified by the shading proceeding up and to the right, the stemidentified by the shading proceeding up and to the right, and the answeridentified by the cross hatching.

FIG. 6G shows an embodiment of a sequential answer format SmartItem ofthe invention as it will be displayed during a test with the variableidentified by the shading proceeding up and to the right, the stemidentified by the shading proceeding up and to the right, and the answeridentified by the cross hatching.

FIG. 6H shows an embodiment of a pictorial entry format SmartItem of theinvention as it will be displayed during a test with the variableidentified by the shading proceeding up and to the right, the stemidentified by the shading proceeding up and to the right, and the answeridentified by the cross hatching.

DETAILED DESCRIPTION

Various embodiments of the invention are described in detail below.Although specific implementations are described, it should be understoodthat this is for illustration purposes only. A person skilled in therelevant art will recognize that other components and configurations canbe used without parting from the spirit and scope of this disclosure.

The invention provides a solution for the present need in the testadministration industry for systems and methods useful for creating,administering, and scoring dynamic, secure smart tests. The inventionsolves the prior art problems by using a computer-based test platformthat is specially programmed to deliver unique, discrete,technologically-enhanced test questions for analyzing a test taker'sknowledge and comprehension of a tested skill. While the tested skill isidentical for each test taker, the test questions used to test the skillare systematically formulated and presented, in real time duringadministration of the test, in pairings of questions, or statements, andanswer alternatives, which are entirely unique to each individualtest-taker. One solution provided by the invention is testing security.

Each test question is formulated to comprise a question stem, code,and/or other design feature used to generate in real time duringadministration of the test, a test question or item having multiplevariations. The test question is used to test an entire skill, ratherthan a simple slice, or part of, a skill. For example, if a student seesa similar test question twice in a row to test a skill, e.g., adding twodouble-digit numbers, then the first presentation of the test questioncan be randomly formulated as: “25+71=?”, or it might be: “17+63=?”. Alltwo-digit numbers (from 10 to 99) are available to be randomly selectedto appear in the first position before the + sign, and in the secondposition after the + sign. As a result, test takers must prepare to betested across the entire skill, i.e., adding two double-digit numbers,rather than a single or multiple memorized static items. This meets oneof the primary purposes of education—to learn a skill, rather thanmemorize questions and answers.

The invention provides an exam that purposefully cannot be re-created.Indeed, the traditional test development model requires teachers orsubject matter experts to produce a new single slice question (or item)whenever a new exam is created. Creating, tracking, and analyzing thesesingle items is time-consuming and wasteful to the time of teachers,subject matter experts, or psychometricians (i.e., test experts). Thepotential for a perpetual, smart exam, as provided by the invention,saves time, development costs, and enhances test administrationefficiency and effectivity, all under the umbrella of a secure testingenvironment.

A detailed discussion of the systems and methods of the invention isprovided below. First, a basic system overview is described. Next, astep-by-step approach to create a smart test using question stems andanswer alternatives coded with variables to present discrete and uniquetest questions is outlined. Third, the system components are identified,followed by a description of a cloud computing system. A detailedexample of identifying, creating, administering, grading, and optimizinga smart test and specific examples of question creation is next. Theincorporation of additional parameters to the system is then delineated.Finally, the collection and retention of relevant data is outlined.

System Overview

The system for creating a secure dynamic test to measure an individual'sknowledge or skills is provided. The system comprises:

-   -   a display in communication with a mobile computer device or a        computer device;    -   a software application operating on the mobile computer device        or on the computer device, the software application configured        to request and receive through a wired and/or wireless        communication network from a testing server located at a testing        location or a remote server in communication with the testing        server through the wired and/or wireless communication network;        -   a question stem uploaded by an owner or author of the test            or an employee, contractor, or agent of the owner or author            and stored in a database of the system,        -   a variable adapted to be combined with the question stem to            form a complete question, previously uploaded by the owner            or author of the test or an employee, contractor, or agent            of the owner or author and stored in the database of the            system, and        -   a plurality of responses previously uploaded by the owner or            author of the test or an employee, contractor, or agent of            the owner or author and stored in the database of the            system; and    -   a processor in communication through the wired and/or wireless        communication network with the software application, as well as        the testing server and/or the remote server, the processor is        configured to randomly call up from the database of the system        upon request from the software application:        -   the question stem,        -   the variable, previously linked to the question stem by the            owner or author of the test or an employee, contractor, or            agent of the owner or author and stored in the database of            the system, and        -   the responses, previously linked to the question stem or the            variable by the owner or author of the test or an employee,            contractor, or agent of the owner or author and stored in            the database of the system;    -   whereby the processor is configured to:        -   combine the question stem and the variable to form the            complete question,        -   identify the response linked to both the stem and the            variable to identify an answer to the completed question,            and        -   communicate the completed question, and, optionally, the            answer or a response to the software application;    -   whereby the software application is configured to:        -   display the complete question and optionally the response,            and        -   record the individual's reply.

Disclosed by FIG. 1 is the system that includes a server 100 comprisinga processor aided by memory that communicates with a database(s). Thedatabase(s) contains question stems coded or linked with variables,responses coded or linked with the question stems and/or variables, andsuper-sets of both incorrect and correct responses, as well as othertypes of response modes. The variable(s) are incorporated into thequestion stems to create complete questions. The database(s) can alsoinclude scoring keys, which include the correct answers to all thepossible questions that may be created from the combinations of thequestion stems and various sources of variables in both the stems andthe answer options, if the question type provides options.

The server 100 is in wired or wireless communication with a testauthor's device 110. The test author's device 110 includes a processoraided by memory that communicates with the server 100. The test author'sdevice 110 also includes an author interface to permit the author tointeract with the server 100.

The server 100 is also in wired or wireless communication with a testingdevice 130. The testing device 130 includes a processor aided by memorythat communicates with the server 100. The testing device 130 alsoincludes an testing interface on a display that is adapted to presentquestions to the test taker and record the responses of the test taker.The testing device may then transmit the responses back to the server100.

The author's device includes a processor aided by memory thatcommunicates with the server 100. The author's device 110 also includesan author interface to permit the author to interact with the server100.

When the test taker accesses the system, the system can generate aunique test in real time and transmit test questions to the test taker'scomputer device or mobile computer device for display at the time oftest administration. The complete question is created with stems codedor linked with variables. Potential responses, which can be correct orincorrect depending on the complete question, are also directly uploadedto and stored in the database by an owner, author, or administrator ofthe test, or an employee, contractor, or agent of the owner, author, oradministrator. Such responses can also be coded or linked with the stemand/or the variables. In this regard, incorrect responses (i.e., thosecoded or linked with only the question or the variable) can bedistinguished from the correct responses (i.e., those coded or linkedwith both the question and the variable).

The reply of the test taker may take a number of forms. Such forms maybe random or fixed. For example, a response mode could be randomlyselected from: a selection of multiple choice alternatives such as DOMC,a short typed response, a longer response similar to an essay answer,the re-ordering of a list, the highlighting of a passage within the textof the question, and many other response forms and formats.Alternatively, the writer could select the response mode that makes iteasiest for the test taker to demonstrate his or her ability in theskill.

If the test taker responds incorrectly to the question (i.e., theirreply is not linked with both the stem and variable), then the testtaker can be notified that either: the response was incorrect, or thetest taker will be presented additional opportunities to respond. Forexample, if the test taker identifies a frog as a mammal, then the testtaker can be asked a second time to identify a mammal with a frog againdisplayed to measure whether the test taker was guessing or, in fact,thinks a frog is a mammal. After the test taker responds, he/she can benotified how his/her response was scored (i.e., correct or incorrect).

The information contained on the database related to question stem typescoded with variables, variables as part of answer options, and potentialcorrect and incorrect answer options (if multiple choice or DOMC) can beuploaded by the owner or author of the test or their employee orcontractor and stored in a database of the system. Furthermore, thesystem can be designed to measure how long the test taker views thecomplete question before responding. This can be used to determinewhether the test taker is attempting to cheat, does not understand thequestion, cannot quickly recall the knowledge necessary to answer thequestion, or the question requires significant time to answer. Forexample, a test taker who takes a short time period to read thequestion, but a long time period to answer the question right or wrong,could be flagged as a test taker that could be attempting to cheat.

Step-by-Step Approach

FIG. 3 depicts a non-limiting, exemplary embodiment that includes thesteps of the process of the invention. An example skill (provided belowin Step 1) will be used for each exemplary step.

Step 1—Skill Evaluation 300.

Evaluate the skill to be measured, also called learning or instructionalobjectives, competencies, proficiencies, performances, behaviors,constructs, abilities, traits, interests, or aptitudes. This evaluationis performed differently by the invention in comparison to a traditionaltest because the purpose of the step is to design the rules of aquestion or item (or more than one question or item) to measure theentire or a large portion of the stated skill. Conversely, the intent oftraditional item design aims to measure a single aspect or slice of theskill, as long as it is covered by the skill. For example, the skillcould be: “Know how the amendments to the Constitution and Bill ofRights protect civil liberties.” A traditional design approach could beto create a question or item that measures a person's knowledge of oneof the amendments, e.g., the 14^(th) Amendment, and then to write aquestion or item for the 14^(th) Amendment. In subsequent exams, or forpractice exams, a different amendment would usually be selected. It isclear that the purpose of the stated skill or objective is that thelearner should know all of the amendments relevant to civil liberties,not just a single one. When using the invention to create a dynamicsingle question or item (i.e., SmartItem) that covers the entire skill,the designer can consider the skill very differently. For example, howmany amendments are there that are relevant to civil liberties? How canthe question or item be formulated to randomly ask about each one of therelevant amendments?

Step 2—Stem Design 310.

The designer designs the question stem or item stem. The designer coulduse programming code in the question or item stem, where the initialstatements and actual task is presented, in order to vary the content ofthe stem when it is presented. For example, code could be written toselect randomly an amendment from all relevant amendments. Amendmentnames and numbers could be contained in a list given in the stem code orprovided in a separate file. The code could also be written to select aquestion to ask about the amendment, or randomly selected from a list orfile. So, one version, when presented to a test taker, might ask, “Howdoes Amendment 15 protect the rights of African Americans?” Anotherversion might ask, “Is this how Amendment 4 protects the rights ofwomen?” Such an item, might be able to generate many thousands ofalternative and relevant “stems”. Variables, such as the amendmentnumber can be coded or linked to the stem. Finally, possible responsescan be coded or linked to the stem and/or the variables.

Step 3—Response Design 320.

The designer provides a way for the test taker to respond. This could berandom or fixed. For example, a response mode could be randomly selectedfrom: (1) a selection of multiple choice alternatives, (2) a short typedresponse, (3) a longer response similar to an essay answer, (4) there-ordering of a list, (5) the highlighting of the passage in the textof the Constitution, and (5) any other response forms or formats.Alternatively, the designer could select the response mode that makes iteasiest for the test taker to demonstrate his or her ability in theskill.

Step 4—Score Design 330.

The designer determines the way(s) to score the item, establishes keys,and generates rules for scoring. For example, the designer can code orlink the proposed responses so that the system evaluates the testtaker's reply to determine if it is linked to both the stem and selectedvariable indicating a correct answer, or conversely only linked to oneelement, which would indicate an incorrect answer.

Step 5—Designer Preview 340.

The designer could preview the test question or test item generated bythe system in the same or similar user interface the test taker wouldsee. By previewing the item several times, the designer can detect anyproblems with the design, and then apply the proper fixes.

Step 6—Editor Preview 350.

Reviewers, editors, and other individuals analyze the test question ortest item prepared by the designer. Like the designer, each reviewerneeds to analyze the design and components of the item, compare with theskill, and analyze the item in functional mode with the preview abilityof the software. The reviewers, which can be subject matter experts, canestablish the quality of the item and its congruence with the skill.While this process exists for the traditional item, its purpose isextended for the dynamic question or item stem. The reviewers are notestablishing that the dynamic question or item stem is an instance of askill, but rather that, when given to many students, the dynamicquestion or item stem measures the entire skill.

Step 7—Test Creation 360.

Completed and final questions or items may then be uploaded to a testingdatabase. Such final questions may also be combined into a test andpresented to test takers as properly designed to test the skills. Theresponses of the test-takers may be collected and uploaded to thetesting database for analysis.

Step 8—Response Analysis 370.

Test results provide data that can be analyzed to evaluate the qualityof the dynamic question, or item stem, and its components. Traditionalanalyses will likely be useful, but dynamic questions or items producenew types of data, such as new kinds of responses, latencies to thoseresponses, orders of events occurring during the item, and other data.New dynamic questions or items based analyses can be used to obtain thisdata.

Step 9—Test Revisions 380.

As a result of the analyses, changes to the content or design of thedynamic questions or item stems can be used to optimize the testquestions or test items. As a result, the dynamic questions or items canbe revised to update and prepare them for continued use in the tests.

Step 10—Skill Evaluation Revisions 390.

Studies can also be conducted routinely to determine or confirm thesuitability of the descriptions of the skills used, the ability of thedynamic question or item to solve problems of validity, achieve bettermeasurement accuracy, better security, greater fairness, and/or lowercost.

Rules Based Dynamic Variable-Answer Identification

FIG. 4 depicts one embodiment of the system in which the systemautomatically identifies variables to link with the stem. In such anembodiment, an author or owner of a test or an employee, contractor, oragent of the owner or author uploads subject rules and question stems410 and stores them in the database of the system. The system evaluateswhether the stems comply with the rules 420. If the stems do not complywith the rules, then the author or owner is notified of thenon-compliant stems. Conversely, if the stems comply with the rules,then the system automatically searches databases or websites previouslyidentified by the author or owner of the test or an employee,contractor, or agent of the owner or author for variables 430. Thesystem determines whether the variables comply with the uploaded rules440. Variables that comply with the rules are stored and variables thatdo not comply with the rules are discarded. Stored variables arecombined with the stems to create complete questions 450. The systemthen searches databases or websites previously identified by the authoror owner for responses to the complete questions 460. The systemdetermines if the identified responses are correct 470. Incorrectresponses can be discarded or stored on the database. In contrast,correct responses are all stored on the database and linked to therelevant stem and variable combinations 480. In this way, the system canautomatically populate the database with the necessary variables andanswers based on rules uploaded by the owner or author of the test.

FIGS. 5A and 5B depicts one embodiment of the system in which the authoror owner of a test, or an employee, contractor, or agent of the owner orauthor codes or links stems, variables, and responses so that the sameskill set may be tested using random question formats. In such anembodiment, an author or owner of a test or an employee, contractor, oragent of the owner or author first creates a skill template in adatabase to match a desired skill capability (e.g., adding two digitnumbers, knowledge of the bill of rights, patentability analysis, etc.)510. This template links stems, variables, and responses. To assist withtemplate creation, a previously-created graphical user interface 512 ora coding interface 514 may be used. The choice of such interfaces maydepend on whether or not the author or owner of a test or an employee,contractor, or agent of the owner or author has coding experience. Afterthe template is created and the responses, stem, and variables arelinked, the author or owner of a test or an employee, contractor, oragent of the owner or author codes or links the question formats thatmay utilize the template and be presented to the test-taker for response520. Responses to the questions may take the form of selecting from alist of choices, creating short-answers, writing essays, drop-and-drag,list building, or performing simulations, which are evaluated by theprocessor. Such evaluation may include the use of computer vision. Next,the system identifies correct responses to stem variable combinationsbased on a determination, if the response is linked to both the questionand variable (i.e., a correct response), versus the response beinglinked only to the stem or to the variable (i.e., an incorrect response)530. Optionally, the author or owner of a test or an employee,contractor, or agent of the owner or author may restrict certain linkedquestion formats from being presented to a certain class of test takers540. For example, a test-taker who is blind should not be presented witha hot area question format. When the system is launched, the softwaredetermines if the testing device supports real-time generation ofdynamic questions 550. If the testing device supports real-time questiongeneration, at the time of question delivery, then the system requests asingle stem and variable 554 and combines those elements and displaysthe randomly-generated question to the test taker 556. Conversely, ifthe testing device does not supports real-time question generation, atthe time of test delivery, then the system requests a plurality of stemsand variables and combines those stems and variables to form a questionpool 552. Such pool of items can then be used to create traditionaltests. For example, a randomly-selected question from the question poolmay be presented to the test taker for a response.

System Components

FIG. 2 depicts a non-limiting embodiment of the system. Such anembodiment includes a general-purpose computing device, including aprocessing unit (CPU or processor) 200, and a system bus 210 thatcouples various system components including the system memory 220 suchas read only memory (ROM) and random access memory (RAM) to theprocessor 200. The system can include a storage device 230 connected tothe processor 200 by the system bus 210. The system can includeinterfaces 240 connected to the processor 200 by the system bus 210. Thesystem can include a cache 202 of high speed memory connected directlywith, in close proximity to, or integrated as part of, the processor200. The system can copy data from the memory 220 and/or a storagedevice 230 to the cache 202 for quick access by the processor 200. Inthis way, the cache 202 provides a performance boost that avoidsprocessor delays, while waiting for data. These and other modules storedin the memory 220, the storage device 230, or the cache 202 can control,or be configured to control, the processor 200 to perform variousactions. Other system memory 220 can be available for use as well. Thememory 220 can include multiple different types of memory with differentperformance characteristics.

Computer Processor

The invention can operate on a computing device with more than oneprocessor or on a group or cluster of computing devices networkedtogether to provide greater processing capability. The processor caninclude any general purpose processor and a hardware module or softwaremodule, stored in an external or internal storage device, configured tocontrol the processor, as well as a special-purpose processor wheresoftware instructions are incorporated into the actual processor design.The processor may essentially be a completely self-contained computingsystem, containing multiple cores or processors, a bus, memorycontroller, cache, etc.

For clarity purposes, a non-limiting, exemplary embodiment of the systemis presented as including individual functional blocks includingfunctional blocks labeled as a “processor”. The functions such blocksrepresent may be provided through the use of either shared or dedicatedhardware, including, but not limited to, hardware capable of executingsoftware and hardware, such as a processor, that is purpose-built tooperate as an equivalent to software executing on a general purposeprocessor. For example, the functions of one or more processors can beprovided by a single shared processor, or multiple processors. Use ofthe term “processor” should not be construed to refer exclusively tohardware capable of executing software. Illustrative embodiments includemicroprocessor and/or digital signal processor (DSP) hardware, read-onlymemory (ROM) for storing software performing the operations discussedbelow, and random access memory (RAM) for storing results. Very largescale integration (VLSI) hardware embodiments, as well as custom VLSIcircuitry in combination with a general purpose DSP circuit, can also beused in the systems of the invention.

System Bus

The system bus can be any of several types of bus structures including amemory bus or memory controller, a peripheral bus, and a local bus usingany of a variety of bus architectures. A basic input/output (BIOS)stored in ROM or the like, can provide the basic routine that helps totransfer information between elements within the computing device, suchas during start-up.

Storage Device

The computing device can further include a storage device such as a harddisk drive, a magnetic disk drive, an optical disk drive, a solid statedrive, a tape drive, or the like. Similar to the system memory, astorage device can be used to store data files, such as locationinformation, menus, software, wired and wireless connection information(e.g., information that may enable the mobile device to establish awired or wireless connection, such as a USB, Bluetooth or wirelessnetwork connection), and any other suitable data. Specifically, thestorage device and/or the system memory can store code and/or data forcarrying out the disclosed techniques among other data.

In one aspect, a hardware module that performs a particular functionincludes the software component stored in a non-transitorycomputer-readable medium in connection with the necessary hardwarecomponents, such as the processor, bus, display, and so forth, to carryout the function. The basic components are known to those of skill inthe art and appropriate variations are contemplated depending on thetype of device, such as whether the device is a small, handheldcomputing device, a desktop computer, or a computer server.

Although a non-limiting, exemplary embodiment described herein employscloud computing and cloud storage, it should be appreciated by thoseskilled in the art that other types of computer-readable media, whichcan store data that are accessible by a computer, such as magneticcassettes, flash memory cards, digital versatile disks, cartridges,random access memories (RAMS), read only memory (ROM), a cable orwireless signal containing a bit stream and the like, can also be usedin the operating environment of the system. Furthermore, non-transitorycomputer-readable storage media, as used herein, can include allcomputer-readable media, with the sole exception being a transitorypropagating signal per se.

Interface

To enable user interaction with the computing device, an input devicerepresents any number of input mechanisms, such as a microphone forspeech, a touch-sensitive screen for gesture or graphical input,keyboard, mouse, motion input, speech, and so forth. An output devicecan also be one or more of a number of output mechanisms known to thoseof skill in the art such as a display screen, speaker, alarm, and soforth. In some instances, multimodal systems enable a user to providemultiple types of input to communicate with the computing device. Thecommunications interface generally governs and manages the user inputand system output. Furthermore, one interface, such as a touch screen,can act as an input, output, and/or communication interface.

There is no restriction on the system operating on any particularhardware arrangement. Therefore, the basic features described herein canbe easily substituted for improved hardware or firmware arrangements.

Software Operations

The logical operations of the various embodiments disclosed areimplemented as: (1) a sequence of computer-implemented steps,operations, or procedures running on a programmable circuit within ageneral use computer, (2) a sequence of computer-implemented steps,operations, or procedures running on a specific-use programmablecircuit; and/or (3) interconnected machine modules or program engineswithin the programmable circuits. The system can practice all or part ofthe recited methods, can be a part of the recited systems, and/or canoperate according to instructions in the recited non-transitorycomputer-readable storage media. Such logical operations can beimplemented as modules configured to control the processor to performparticular functions according to the programming of the module. Forexample, if a storage device contains modules configured to control theprocessor, then these modules can be loaded into RAM or memory atruntime, or may be stored in other computer-readable memory locations.Certain embodiments can utilize any of the following programminglanguages: Python, Flask, jinja2, RactJS, JavaScript, Express, VM2, andReactJS. Having disclosed some components of a computing system, thedisclosure now turns to a description of cloud computing, which is apreferred operating environment of the invention.

Cloud System

Cloud computing is a type of Internet-based computing in which a varietyof resources are hosted and/or controlled by an entity and madeavailable by the entity to authorized users via the Internet. A cloudcomputing system is typically configured so that a variety of electronicdevices can communicate via a network for purposes of exchanging contentand data. The system of the invention can be configured for use on awide variety of network configurations that facilitate theintercommunication of electronic devices. For example, each of thecomponents of a cloud computing system can be implemented in a localizedor distributed fashion in a network.

Cloud Resources

The cloud computing system can be configured to include cloud computingresources (i.e., “the cloud”). Cloud resources can include a variety ofhardware and/or software resources, such as cloud servers, clouddatabases, cloud storage, cloud networks, cloud applications, cloudplatforms, and/or any other cloud-based resources. In some cases, thecloud resources are distributed. Cloud storage can include multiplestorage devices. In some cases, cloud resources can be distributedacross multiple cloud computing systems and/or an individualnetwork-enabled computing devices. Cloud computing resources cancommunicate with a server, a database, and/or any other network-enabledcomputing device to provide the cloud resources.

In some cases, the cloud resources can be redundant. For example, ifcloud computing resources are configured to provide data backupservices, then multiple copies of the data can be stored such that thedata is still available to the user even if a storage resource isoffline, busy, or otherwise unavailable to process a request. In anotherexample, if a cloud computing resource is configured to providesoftware, then the software can be available from different cloudservers so that the software can be served from any of the differentcloud servers. Algorithms can be applied such that the closest server orthe server with the lowest current load is selected to process a givenrequest.

User Terminals

A user interacts with cloud computing resources through user terminalsor testing devices connected to a network by direct and/or indirectcommunication. Cloud computing resources can support connections from avariety of different electronic devices, such as servers; desktopcomputers; mobile computers; handheld communications devices (e.g.,mobile phones, smart phones, tablets); set top boxes; network-enabledhard drives; and/or any other network-enabled computing devices.Furthermore, cloud computing resources can concurrently acceptconnections from and interact with multiple electronic devices.Interaction with the multiple electronic devices can be prioritized oroccur simultaneously.

Cloud computing resources can provide cloud resources through a varietyof deployment models, such as public, private, community, hybrid, and/orany other cloud deployment model. In some cases, cloud computingresources can support multiple deployment models. For example, cloudcomputing resources can provide one set of resources through a publicdeployment model and another set of resources through a privatedeployment model.

In some configurations, a user terminal can access cloud computingresources from any location where an Internet connection is available.However, in other cases, cloud computing resources can be configured torestrict access to certain resources such that a resource can only beaccessed from certain locations. For example, if a cloud computingresource is configured to provide a resource using a private deploymentmodel, then the cloud computing resource can restrict access to theresource, such as by requiring that a user terminal access the resourcefrom behind a firewall.

Service Models

Cloud computing resources can provide cloud resources to user terminalsthrough a variety of service models, such as Software as a Service(SaaS), Platform as a Service (PaaS), Infrastructure as a Service(IaaS), and/or any other cloud service models. In some cases, cloudcomputing resources can provide multiple service models to a userterminal. For example, cloud computing resources can provide SaaS andIaaS to a user terminal. Cloud computing resources can provide differentservice models to different user terminals. For example, cloud computingresources can provide SaaS to one user terminal and PaaS to another userterminal.

User Interaction

In some cases, cloud computing resources can maintain an accountdatabase. The account database can store profile information forregistered users and/or test takers. The profile information can includeresource access rights, such as software the user and/or test taker ispermitted to use. The profile information can also include usageinformation, such as computing resources consumed, data storagelocation, security settings, personal configuration settings, etc. Insome cases, the account database can reside on a database or serverremote to cloud computing resources, such as, servers or databases.

Cloud computing resources can provide a variety of functionality thatrequires user interaction. Accordingly, a user interface (UI) can beprovided for communicating with cloud computing resources and/orperforming tasks associated with the cloud resources. The UI can beaccessed via an end user terminal in communication with cloud computingresources. The UI can be configured to operate in a variety of clientmodes, including a fat client mode, a thin client mode, or a hybridclient mode, depending on the storage and processing capabilities ofcloud computing resources and/or the user terminal. A UI can beimplemented as a standalone application operating at the user terminalin some embodiments. In other embodiments, a web browser-based portalcan be used to provide the UI. Any other configuration to access cloudcomputing resources can also be used in the various embodiments.

Example

This example is provided for the purpose of illustration only, and thesubject matter is not limited to this example, but rather encompassesall variations which are evident as a result of the teaching providedherein. The example can be implemented with regard to a testadministered at a testing location.

The testing location includes a general-purpose computing device,software application, an Internet access point, and/or at least onedatabase. The general-purpose computing device can be a portablepersonal device, such as, a mobile phone, smart phone, or tablet. TheInternet access point can be a wireless access point.

Account Login

The individual begins by visiting a website or loading a computerapplication. A server determines if this is the first visit by thegeneral-purpose computing device. If the server determines that this isa subsequent visit by the general-purpose computing device, then priorgeneral information (e.g., name, contact information, test to beadministered, etc.) is loaded. If this is the first visit by thegeneral-purpose computing device, then relevant information (e.g., name,contact information, test to be administered, etc.) is collected, or thetest taker may proceed with the test.

Determination of Skills to be Tested

After the individual logs into an account or selects a test to beadministered, the skills to be tested are identified. Such skills may beidentified by the individual testing program, a certified testingauthority (e.g., the Association of American Medical Colleges whoadminister the Medical College Admission Test®), or a third party. Thelevels of the skills to be tested can be updated as the individualproceeds through the test, as in a computerized adaptive test. Theskills to be tested can be updated on a set schedule, or in real timeduring test administration.

Creation of Smart Questions

Once the skills to be tested are determined, smart questions (i.e.,SmartItems) are created. These smart questions are a combination of aquestion stem combined with at least one variable tied to the partialquestion stem, both of which are stored on a database and accessed by aprocessor of the system. The question stems, based directly on theskills to be measured, are selected at random. This way a question stemcan be presented as a test question, first, for one test taker, whereasthe same partial question stem can be presented last to another testtaker. After the partial question stem is selected, at least onevariable tied to the partial question stem is selected at random. Thevariable is then combined with the partial question stem to create thetest question or test item, i.e., smart question or smart item. Withenough stems and variables, along with SmartItem features, suchrandomization can provide that no two tests will ever be the same.Furthermore, hacking the database to obtain all the question stems andvariables provides no benefit because the questions are randomly createdand presented. Knowing the contents of the database provides no hints asto how questions will be formatted or presented to the test taker.

Identification of the Answer

Once the smart question is created, the answer is supplied by the testtaker or identified by the test taker. All of the potential responses toall the potential questions can be stored in one database, or onmultiple databases. Once the smart question is finalized (i.e., thevariables are combined with the partial question stem), the specificcorrect and incorrect answer to the unique smart question can beidentified by determining which of the potential responses are linked toboth the stem and the variable. Again, these potential responses arestored on a database and accessed by a processor. Conversely, a replycan be supplied by the test taker in a short-answer text box or as anessay response and compared to potential responses, and the reply can bescored based on whether it more closely matches a stored response linkedto both the stem and variable (i.e., a correct answer) or a responselinked to only one of the stem or of the variable (i.e., an incorrectanswer).

In identifying the answer, additional related correct and incorrectanswer options can also be selected at random to be presented to thetest taker.

Progression of Question/Answer

Once the question is identified, a response is required of the testtaker. Indeed, responses to the smart question can take many forms. Forexample, the smart question may: (1) require a selection of multiplechoice alternatives, one of which is the answer, and the rest of whichare incorrect answer options, (2) require a short typed response whichmust incorporate certain relevant words to be deemed correct, (3)require a longer response similar to an essay answer, (4) require there-ordering of a list wherein the answer is the correct order of thelist, or (5) require the highlighting of a specific passage in the smartquestion.

Such unique response requirements provide an added layer of security forthe test and can also protect against cheating. The test taker'sresponses are transmitted back to the database to be stored and can besubmitted for further analysis by the system.

Collection of Data

In some configurations, during the testing described above, a storagedevice or resource can be used to store relevant data transmitted fromthe test taker's device to a database over a wired or wirelesscommunication network. For example, how long it takes to read andprocess the question stem or item stem can be captured, along with thelatencies for responses to any option presented or to supply a response,or to record the order that events occurred when each question waspresented. The data stored can be incorporated into the disclosed systemand methods to refine the testing experience or evaluate security. Inaddition, collected data can be used to evaluate smart questionsdescribed above. As a result, smart questions can serve more than onepurpose, e.g., quality of measurement, better security, more fairness,quality of teaching instruction/curriculum, and others. Those purposescan be routinely evaluated.

Question Formats

Multiple Choice Format

FIG. 6A shows a non-limiting embodiment of a type of question of theinvention. In this non-limiting, exemplary format, the stem and variableare combined to create a complete question. Next, a response linked toboth the stem and variable (i.e., a correct answer) along with at leastone additional response linked to either the stem or variable (i.e., anincorrect answer) are presented to the test taker. Such presentation mayoccur sequentially (i.e., one at a time) or all at once. The testtaker's reply takes the form of selecting if a proposed response iscorrect or incorrect, or selecting one of the plurality of potentialresponses displayed.

Text Entry Format

FIG. 6B shows a non-limiting embodiment of a type of question of theinvention. In this non-limiting, exemplary format, the stem and variableare combined to create a complete question. Next, a response linked toboth the stem and variable (i.e., a correct answer) is identified by thesystem, but not presented to the test taker. The test taker's replytakes the form of entering letters or numbers into a dialogue box. Thetest taker's reply is then compared to the identified answer todetermine if the test taker's reply is correct.

Boolean Answer Format

FIG. 6C shows a non-limiting embodiment of a type of question of theinvention. In this non-limiting, exemplary format, the stem and variableare combined to create a complete question. Next, a Boolean response(e.g., “Yes” or “No”, or “True” or “False”) linked to both the stem andvariable is identified by the system, but not presented to the testtaker. The test taker's reply takes the form of selecting one of theBoolean responses. The test taker's reply is then compared to theidentified answer to determine if the test taker's reply is correct.

Jeopardy Approach

FIG. 6D shows a non-limiting embodiment of a type of question of theinvention. In this non-limiting, exemplary format, the manner by whichthe test questions or test items and answer options are presented to thetest taker are reversed. Specifically, the answer and stem can bepresented to the test taker, who must then determine what variable islinked to the answer and question.

Hot Area Format

FIG. 6E shows a non-limiting embodiment of a type of question of theinvention. In such a non-limiting exemplary format, the stem or variablecan be an image with specific areas of the image linked to differentstems or variables. The test taker's reply takes the form of selectingan area of an image, and the system determines if the area identified bythe test taker is linked to both the variable and the stem.

List Format

FIG. 6F shows a non-limiting embodiment of a type of question of theinvention. In such a non-limiting, exemplary format, multiple responsescan be linked to the same stem and variable. As a result, there will bemore than one correct answer. For such a format, a stem and a variablecan be combined to form a question, and the test taker's reply takes theform of a number of numerical and/or textual inputs. Each input isevaluated to determine if that reply matches a correct answer, within arange of tolerance (e.g., spelling).

Sequential Format

FIG. 6G shows a non-limiting embodiment of a type of question of theinvention. In such a non-limiting, exemplary format, a stem and variableare combined to form a question. The test taker is then presented with asequence of possible responses and must select whether the listedresponse is correct or not. In such a format, the responses presented tothe test taker can cease as soon as the test taker identifies a correctanswer or may continue to present additional responses for the testtaker's evaluation.

Pictorial Format

FIG. 6H shows a non-limiting embodiment of a type of question of theinvention. In such a non-limiting, exemplary format, a stem and variableare combined to form a question. The test taker's reply must then takethe form of selecting a relevant picture or image. The reply is thencompared to responses linked to both the stem and variable, and if thereis a match, within a range of tolerance, the answer is scored.

Incorporation of Additional Parameters

Series Learning Approach

In an exemplary, non-limiting embodiment of the invention, partialquestion stems can be tied to additional and sequential question stemsto create a series testing approach. Such a linkage may permit for thecreation of a unique series of questions. For example, the skill to betested can be the first law of thermodynamics. Before one can evaluatethe first law of thermodynamics, however, an individual must make suretheir units are correct. A smart question, for example, on the first lawof thermodynamics can first relate to unit conversion, (e.g., convertingall stated measurements into SI units). A second smart question canbuild on this concept and specifically test the test taker's knowledgebased on SI units. The test taker's actual knowledge with regard to thefirst law of thermodynamics and how it is applied is measured. Such anapproach provides a more complete measurement of a test taker'sknowledge and cognitive function.

While this subject matter has been disclosed with reference to specificembodiments, it is apparent that other embodiments and variations can bedevised by others skilled in the art without departing from the truespirit and scope of the subject matter described herein. The appendedclaims include all such embodiments and equivalent variations.

What is claimed is:
 1. A system for creating a secure dynamic test tomeasure a test taker's knowledge or cognitive function, the systemcomprising: a display in communication with a computer device; asoftware application operating on the computer device, the softwareapplication configured to request and receive through a wired and/orwireless communication network from a testing server in communicationwith a database, wherein the testing server is located at a testinglocation or communicates with a remote server through the wired and/orwireless communication network: a question stem uploaded into thedatabase, a variable configured to be combined with the question stem toform a complete question, the variable also uploaded into the database,and a plurality of responses also uploaded into the database; and aprocessor in communication through the wired and/or wirelesscommunication network with the software application, as well as thetesting server and/or remote server, the processor is configured torandomly call up from the database upon request from the softwareapplication: the question stem, the variable uploaded into the database,and the responses uploaded into the database; whereby the processor isconfigured to: combine the question stem and the variable to form thecomplete question, identify the response linked to both the stem and thevariable to identify an answer to the completed question, andcommunicate the completed question and optionally the answer or aresponse to the software application; whereby the software applicationis configured to: display the complete question and optionally theresponse, and record the answer or the response.
 2. The system of claim1 wherein the software application is further configured to request andreceive the answer or the response through a wired and/or wirelesscommunication network from the computer device.
 3. The system of claim 2wherein the processor is further configured to compare the answer or theresponse to the answer and communicate a match to the softwareapplication.
 4. The system of claim 3 wherein the processor is furtherconfigured to tally the number of matches between each of the answer orthe response, and communicate the tally to the software application. 5.The system of claim 3 wherein the software application is furtherconfigured to display when the answer or the response matches theanswer.
 6. The system of claim 1 wherein, after a time limit set in thedatabase for a question stem to be answered, the software applicationstops recording the answer or the response.
 7. The system of claim 1wherein the processor is configured to communicate the answer or theresponse to the software application, and the software application isconfigured to display the response.
 8. The system of claim 1 wherein thequestion stem, variable, or answer is an image without letters ornumbers.
 9. The system of claim 1 wherein the response is a specificarea of an image uploaded into the database.
 10. A system for creating asecure dynamic test to measure a test taker's knowledge or cognitivefunction, the system comprising: a display in communication with acomputer device; a software application operating on a websiteaccessible through a wired or wireless communications network by thecomputer device, the software application configured to request andreceive through the wired and/or wireless communication network from atesting server located at a testing location or a remote server incommunication with the testing server through the wired and/or wirelesscommunication network: a question stem stored in a database incommunication with the testing server, a variable adapted to be combinedwith the question stem to form a complete question stored in thedatabase, a plurality of responses stored in the database, and aprocessor in communication through the wired and/or wirelesscommunication network with the software application, as well as thetesting server and/or remote server, the processor is configured to callup randomly from a database of the system upon request from the softwareapplication: the question stem, the variable stored in the database, andthe responses linked to the question stem or the variable stored in thedatabase; whereby the processor is configured to: combine the questionstem and the variable to form the complete question, identify theresponse linked to both the stem and the variable to identify an answerto the completed question, and communicate the completed question, andoptionally the answer or a response to the software application; wherebythe software application is configured to: display the complete questionand optionally the response, and record the response and/or the answer.11. The system of claim 10 wherein the software application is furtherconfigured to request and receive the response and/or the answer througha wired and/or wireless communication network from the computer device.12. The system of claim 11 wherein the processor is further configuredto compare the response and/or the answer to the question andcommunicate a match to the software application.
 13. The system of claim12 wherein the processor is further configured to tally the number ofmatches between the response and/or the answer, and communicate thetally to the software application.
 14. The system of claim 12 whereinthe software application is further configured to display when theresponse and/or the answer matches the answer in the database.
 15. Thesystem of claim 10 wherein, after a time limit stored in the databasefor a question stem to be answered, the software application stopsrecording the test taker's answer.
 16. The system of claim 10 whereinthe processor is configured to communicate the response and/or theanswer to the software application, and the software application isconfigured to display the response and/or the answer.
 17. The system ofclaim 10 wherein the stem, variable, or answer is an image withoutletters or numbers.
 18. The system of claim 10, wherein the response isa specific area of an image stored in the database.
 19. A method forcreating a secure dynamic test to measure a test taker's knowledge orcognitive skills, the method comprising: receiving a request for aquestion or question stem from a testing server or a remote serverplaced using a software application operating on a computer device, andwherein the computer device communicates through a wired and/or wirelesscommunication network with the testing server at a testing location orwith the remote server in a location that is remote to the testinglocation and in communication with the testing server; upon receivingthe request for a question, using a processor to randomly call up from adatabase in communication with the testing server: a question stemstored in the database, a variable adapted to be combined with thequestion stem to form a complete question and stored in the database,and a plurality of responses stored in the database and linked to thequestion stem or variable; optionally, combining the question stem andthe variable to form a complete question; using the processor toidentify the response linked to both the question stem and the variableto identify the answer to the question stem; and transmitting thequestion stem or complete question, the answer, and optionally at leastone additional response to the software application.
 20. A method forcreating a secure dynamic test to measure a test taker's knowledge orcognitive function, the method comprising: receiving a request for aquestion with a testing server or a remote server placed through awebsite accessible using a unique registered user access credential,whereby the request is placed using a software application operating ona computer device also operating the software application, and whereinthe computer device communicates through a wired and/or wirelesscommunication network with the testing server at a testing location orwith the remote server in a location that is remote to the testinglocation in communication with the testing server; upon receiving therequest for a question, using a processor to randomly call up: (1) aquestion or a question stem; (2) optionally a variable that can becombined with the question stem to form a complete question, and (3) aplurality of correct and incorrect options to the question or questionstem; optionally, combining the question stem and the variable to form acomplete question; identifying a correct answer to the question stem orcompleted question from a database using the processor; and transmittingthe question stem or complete question, the answer, and optionally atleast one incorrect option to the question stem or complete question tothe software application.